Papers with pre-training tasks modeling
Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models can be fine-tuned on domain-specific unlabeled data . however, most further pre-training works just keep running the conventional pre- training task . |
| Approach: | They propose to add a further pre-training phase to the model to improve downstream tasks . they propose to use a domain-adaptive pre-tuning phase to fine-tune the models on unlabeled data . |
| Outcome: | The proposed method improves multiple task-oriented dialogue downstream tasks. |